Software Topological Message Routing and Aggregation Techniques for Large Scale Parallel Systems

Supercomputing networks are designed to minimize message latency. The focus on low latency implies a best effort attempt to deliver each message injected onto the network as soon as possible. Ostensibly, prioritizing message latency is important, and there are numerous examples of applications that benefit from it, but while this latency-centric view of a network seems logical, it often leads to unintended consequences. It may at first appear surprising that one would find fault with the idea of trying to deliver messages as soon as possible, but if we take a broader view of communication and consider modern world systems for delivery and transportation, we will find that the micro-management of the latency-centric approach is hardly the norm. As an example, consider an urban rail transit system. Traveling by train requires being aware of the schedule and sharing travel space with other passengers. The time to reach the destination may be higher on average than by car, as it involves waiting at the station and changing trains if the destination does not happen to lie along the route of the initially boarded train. Unless one is traveling in a densely populated area, a train may not even be an option. The various restrictions and limitations, which are a source of inconvenience for the passengers, also allow trains to function more efficiently. In many ways a public transit system is an example of a system that deemphasizes latency, that is each individual’s travel time, to improve bandwidth, the number of people transported to their destination per unit time. Compared to the bandwidth-centric approach of a rail system, roads and highways are decidedly latencycentric. Driving a car involves few of the limitations of a public transit system, and under normal conditions leads to significantly shorter travel times. Despite the higher convenience of cars, public transportation remains an indispensable piece of infrastructure in large cities. Allowing large numbers of people to travel using a minimal real estate footprint makes public transit systems a key prerequisite for supporting dense population concentrations of large cities, where office and living space are stacked vertically, while transportation is often limited to a fraction of the space along the two dimensions of the street level. If instead of using public transportation, everyone chose to drive to work in these areas, traffic congestion would increase travel times several fold, and other than the very earliest travelers, most people upon arrival at their destination would find that they have nowhere to park. As latency-optimized systems, supercomputing networks are designed like highways for the data traveling between the increasingly densely populated computing resources of a large system. This work explores the idea of reorganizing the network in software to act more like the public transportation system in order to significantly improve the efficiency of handling a high volume of communication. Toward this goal, we introduce the Topological Routing and Aggregation Module (TRAM), a software communication system for efficiently handling high volume communication patterns, particularly ones involving a high volume of fine-grained communication.